OCPBUGS-59734: fix(azure): resolve credential caching issues around UAMI support #1238

bryan-cox · 2025-08-01T14:48:26Z

Summary

This PR fixes credential caching issues in Azure storage operations and adds caching support for User Assigned Managed Identity (UAMI) credentials.

Caching at the driver level was not enough so a global cache was introduced so that we are not getting new credentials over and over from Azure.

Changes

fix(azure): fix credential caching key mismatch in driver storageAccountsClient
Resolves credential caching key inconsistencies in the storage accounts client
fix(azure): fix credential caching key mismatch in azureclient
Fixes credential caching key mismatches in the Azure client implementation
feat(azure): add ensureUAMICredentials function with comprehensive tests
- Adds new ensureUAMICredentials function to obtain and cache Azure TokenCredential using User Assigned Managed Identity (UAMI)
- Function loads credentials from global cache or creates new ones with proper environment configuration
- Comprehensive unit tests covering environment variable handling, cache behavior, and error scenarios
- Tests follow table-driven pattern and use t.Setenv for proper environment handling
- Updates Azure client instantiation to use UAMI credentials when available

Testing

Added comprehensive unit tests for the new ensureUAMICredentials function
Tests follow established codebase patterns and conventions

Impact

Improves credential management reliability in Azure storage operations
Enables proper UAMI support for managed Azure environments
No breaking changes to existing functionality

sjenning · 2025-08-01T15:52:00Z

/lgtm

bryan-cox · 2025-08-01T17:38:13Z

/test hypershift-e2e-aks

bryan-cox · 2025-08-01T19:29:41Z

/test hypershift-e2e-aks

openshift-ci-robot · 2025-08-04T16:35:47Z

@bryan-cox: This pull request references Jira Issue OCPBUGS-60103, which is invalid:

expected the bug to target the "4.20.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

Use consistent azureCredentialsKey for both storing and loading cached credentials instead of mixing azureCredentialsKey and userAssignedIdentityCredentialsFilePath. This prevents repeated credential recreation and eliminates redundant log messages.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

…untsClient Similar to the azureclient fix, use consistent azureCredentialsKey for both storing and loading cached credentials instead of mixing azureCredentialsKey and userAssignedIdentityCredentialsFilePath. This prevents repeated credential recreation in the Azure driver's storageAccountsClient method.

Use consistent azureCredentialsKey for both storing and loading cached credentials instead of mixing azureCredentialsKey and userAssignedIdentityCredentialsFilePath. This prevents repeated credential recreation in the Azure driver's storageAccountsClient method. Signed-off-by: Bryan Cox <[email protected]>

bryan-cox · 2025-08-04T18:30:35Z

/test hypershift-e2e-aks

bryan-cox · 2025-08-07T09:04:40Z

/test hypershift-e2e-aks

bryan-cox · 2025-08-07T10:54:30Z

/test hypershift-e2e-aks

bryan-cox · 2025-08-07T13:50:38Z

/test hypershift-e2e-aks

flavianmissi · 2025-08-07T15:09:22Z

Hey @bryan-cox! Can you elaborate on your decision of caching the entire storage driver, vs for example caching the credentials within the driver?

bryan-cox · 2025-08-07T15:33:16Z

Hey @bryan-cox! Can you elaborate on your decision of caching the entire storage driver, vs for example caching the credentials within the driver?

@flavianmissi it's still in WIP so I wouldn't consider what is here to be the final solution.

flavianmissi · 2025-08-08T12:03:06Z

Understood, I'll hold my horses 🤠

openshift-ci-robot · 2025-08-12T09:51:54Z

@bryan-cox: This pull request references Jira Issue OCPBUGS-59734, which is invalid:

expected the bug to target the "4.20.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

Use consistent azureCredentialsKey for both storing and loading cached credentials instead of mixing azureCredentialsKey and userAssignedIdentityCredentialsFilePath. This prevents repeated credential recreation and eliminates redundant log messages.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2025-08-12T20:46:04Z

@bryan-cox: This pull request references Jira Issue OCPBUGS-59734, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.20.0) matches configured target version for branch (4.20.0)
bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

No GitHub users were found matching the public email listed for the QA contact in Jira ([email protected]), skipping review request.

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2025-08-12T20:47:38Z

@bryan-cox: This pull request references Jira Issue OCPBUGS-59734, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.20.0) matches configured target version for branch (4.20.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

No GitHub users were found matching the public email listed for the QA contact in Jira ([email protected]), skipping review request.

In response to this:

Summary

This PR fixes credential caching issues in Azure storage operations and adds caching support for User Assigned Managed Identity (UAMI) credentials.

Caching at the driver level was not enough so a global cache was introduced so that we are not getting new credentials over and over from Azure.

Changes

fix(azure): fix credential caching key mismatch in driver storageAccountsClient
Resolves credential caching key inconsistencies in the storage accounts client

fix(azure): fix credential caching key mismatch in azureclient
Fixes credential caching key mismatches in the Azure client implementation

feat(azure): add ensureUAMICredentials function with comprehensive tests

Adds new ensureUAMICredentials function to obtain and cache Azure TokenCredential using User Assigned Managed Identity (UAMI)

Function loads credentials from global cache or creates new ones with proper environment configuration

Comprehensive unit tests covering environment variable handling, cache behavior, and error scenarios

Tests follow table-driven pattern and use t.Setenv for proper environment handling

Updates Azure client instantiation to use UAMI credentials when available

Testing

Added comprehensive unit tests for the new ensureUAMICredentials function

Tests follow established codebase patterns and conventions

Impact

Improves credential management reliability in Azure storage operations

Enables proper UAMI support for managed Azure environments

No breaking changes to existing functionality

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

bryan-cox · 2025-08-13T00:49:21Z

@flavianmissi this is ready for review now 😄

bryan-cox · 2025-08-13T01:27:01Z

The logs show the credentials being stored once then loaded from the cache then on

'''
I0813 00:12:11.956008 1 azure.go:1376] Storing UAMI credentials to global cache
...

I0813 00:21:04.989721 1 azure.go:1348] Loaded UAMI credentials from cache
I0813 00:21:05.603952 1 azure.go:1348] Loaded UAMI credentials from cache
I0813 00:21:26.034294 1 azure.go:1348] Loaded UAMI credentials from cache
I0813 00:21:26.333995 1 azure.go:1348] Loaded UAMI credentials from cache
'''

https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshift_cluster-image-registry-operator/1238/pull-ci-openshift-cluster-image-registry-operator-main-hypershift-e2e-aks/1955410821643243520/artifacts/hypershift-e2e-aks/hypershift-azure-run-e2e/artifacts/TestCreateCluster/namespaces/e2e-clusters-xhkq8-create-cluster-vb2c6/core/pods/logs/cluster-image-registry-operator-f599f47bd-qbwvt-cluster-image-registry-operator.log

bryan-cox · 2025-08-13T09:12:29Z

/test e2e-aws-operator

pkg/storage/azure/azure.go

pkg/storage/azure/azureclient/azureclient.go

- Add ensureUAMICredentials function to obtain and cache Azure TokenCredential using User Assigned Managed Identity (UAMI) - Function loads credentials from global cache or creates new ones with proper environment configuration - Add comprehensive unit tests covering environment variable handling, cache behavior, and error scenarios - Tests use table-driven pattern following codebase conventions and t.Setenv for proper environment handling - Update Azure client instantiation to use UAMI credentials when available

flavianmissi · 2025-08-13T13:30:11Z

/lgtm

openshift-ci · 2025-08-13T13:32:15Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bryan-cox, flavianmissi, sjenning

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [flavianmissi]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

xenolinux · 2025-08-13T14:43:13Z

/label docs-approved

sferich888 · 2025-08-13T15:23:54Z

/label px-approved

bryan-cox · 2025-08-13T16:54:42Z

/test e2e-hypershift-conformance

openshift-ci · 2025-08-13T18:40:59Z

@bryan-cox: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-azure-operator	`6d90565`	link	false	`/test e2e-azure-operator`
ci/prow/okd-scos-e2e-aws-ovn	`6d90565`	link	false	`/test okd-scos-e2e-aws-ovn`
ci/prow/e2e-azure-ovn	`6d90565`	link	false	`/test e2e-azure-ovn`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

xiuwang · 2025-08-14T12:14:06Z

/label qe-approved

openshift-ci-robot · 2025-08-14T12:19:05Z

@bryan-cox: Jira Issue OCPBUGS-59734: All pull requests linked via external trackers have merged:

openshift/cluster-image-registry-operator#1238

Jira Issue OCPBUGS-59734 has been moved to the MODIFIED state.

In response to this:

Summary

This PR fixes credential caching issues in Azure storage operations and adds caching support for User Assigned Managed Identity (UAMI) credentials.

Caching at the driver level was not enough so a global cache was introduced so that we are not getting new credentials over and over from Azure.

Changes

fix(azure): fix credential caching key mismatch in driver storageAccountsClient
Resolves credential caching key inconsistencies in the storage accounts client

fix(azure): fix credential caching key mismatch in azureclient
Fixes credential caching key mismatches in the Azure client implementation

feat(azure): add ensureUAMICredentials function with comprehensive tests

Adds new ensureUAMICredentials function to obtain and cache Azure TokenCredential using User Assigned Managed Identity (UAMI)

Function loads credentials from global cache or creates new ones with proper environment configuration

Comprehensive unit tests covering environment variable handling, cache behavior, and error scenarios

Tests follow table-driven pattern and use t.Setenv for proper environment handling

Updates Azure client instantiation to use UAMI credentials when available

Testing

Added comprehensive unit tests for the new ensureUAMICredentials function

Tests follow established codebase patterns and conventions

Impact

Improves credential management reliability in Azure storage operations

Enables proper UAMI support for managed Azure environments

No breaking changes to existing functionality

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

bryan-cox · 2025-08-14T12:31:34Z

/jira backport release-4.19

openshift-ci-robot · 2025-08-14T12:31:48Z

@bryan-cox: The following backport issues have been created:

OCPBUGS-60517 for branch release-4.19

Queuing cherrypicks to the requested branches to be created after this PR merges:
/cherrypick release-4.19

In response to this:

/jira backport release-4.19

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-cherrypick-robot · 2025-08-14T12:32:44Z

@openshift-ci-robot: new pull request created: #1243

In response to this:

@bryan-cox: The following backport issues have been created:

OCPBUGS-60517 for branch release-4.19

Queuing cherrypicks to the requested branches to be created after this PR merges:
/cherrypick release-4.19

In response to this:

/jira backport release-4.19

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-bot · 2025-08-14T13:38:44Z

[ART PR BUILD NOTIFIER]

Distgit: ose-cluster-image-registry-operator
This PR has been included in build ose-cluster-image-registry-operator-container-v4.20.0-202508141315.p0.gb04c7c1.assembly.stream.el9.
All builds following this will include this PR.

openshift-ci bot requested review from flavianmissi and ricardomaraschini August 1, 2025 14:49

openshift-ci bot assigned sjenning Aug 1, 2025

openshift-ci bot added lgtm Indicates that a PR is ready to be merged. and removed lgtm Indicates that a PR is ready to be merged. labels Aug 1, 2025

bryan-cox marked this pull request as draft August 1, 2025 17:36

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 1, 2025

bryan-cox changed the title ~~fix(azure): fix credential caching key mismatch in UserAssignedIdentityCredentials~~ OCPBUGS-60103: fix(azure): fix credential caching key mismatch in UserAssignedIdentityCredentials Aug 4, 2025

openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Aug 4, 2025

bryan-cox force-pushed the inotify-bug branch from de61ac2 to 3bdc326 Compare August 4, 2025 16:51

bryan-cox force-pushed the inotify-bug branch from 3bdc326 to 4ec2db3 Compare August 4, 2025 18:23

bryan-cox force-pushed the inotify-bug branch from 4ec2db3 to 3002b10 Compare August 7, 2025 09:04

bryan-cox force-pushed the inotify-bug branch from 3002b10 to d772578 Compare August 7, 2025 10:54

bryan-cox force-pushed the inotify-bug branch from d772578 to b334aad Compare August 7, 2025 13:50

bryan-cox changed the title ~~OCPBUGS-60103: fix(azure): fix credential caching key mismatch in UserAssignedIdentityCredentials~~ OCPBUGS-59734: fix(azure): fix credential caching key mismatch in UserAssignedIdentityCredentials Aug 12, 2025

bryan-cox force-pushed the inotify-bug branch from 2cff080 to b958d28 Compare August 12, 2025 23:27

flavianmissi reviewed Aug 13, 2025

View reviewed changes

pkg/storage/azure/azure.go Outdated Show resolved Hide resolved

flavianmissi reviewed Aug 13, 2025

View reviewed changes

pkg/storage/azure/azureclient/azureclient.go Show resolved Hide resolved

bryan-cox force-pushed the inotify-bug branch from b958d28 to 6d90565 Compare August 13, 2025 12:58

openshift-ci bot assigned flavianmissi Aug 13, 2025

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Aug 13, 2025

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 13, 2025

openshift-ci bot added the docs-approved Signifies that Docs has signed off on this PR label Aug 13, 2025

openshift-ci bot added the px-approved Signifies that Product Support has signed off on this PR label Aug 13, 2025

openshift-ci bot added the qe-approved Signifies that QE has signed off on this PR label Aug 14, 2025

openshift-merge-bot bot merged commit b04c7c1 into openshift:main Aug 14, 2025
13 of 16 checks passed

openshift-cherrypick-robot mentioned this pull request Aug 14, 2025

[release-4.19] OCPBUGS-60517: fix(azure): resolve credential caching issues around UAMI support #1243

Merged

OCPBUGS-59734: fix(azure): resolve credential caching issues around UAMI support #1238

OCPBUGS-59734: fix(azure): resolve credential caching issues around UAMI support #1238

Uh oh!

Conversation

bryan-cox commented Aug 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Testing

Impact

Uh oh!

sjenning commented Aug 1, 2025

Uh oh!

bryan-cox commented Aug 1, 2025

Uh oh!

bryan-cox commented Aug 1, 2025

Uh oh!

openshift-ci-robot commented Aug 4, 2025

Uh oh!

bryan-cox commented Aug 4, 2025

Uh oh!

bryan-cox commented Aug 7, 2025

Uh oh!

bryan-cox commented Aug 7, 2025

Uh oh!

bryan-cox commented Aug 7, 2025

Uh oh!

flavianmissi commented Aug 7, 2025

Uh oh!

bryan-cox commented Aug 7, 2025

Uh oh!

flavianmissi commented Aug 8, 2025

Uh oh!

openshift-ci-robot commented Aug 12, 2025

Uh oh!

openshift-ci-robot commented Aug 12, 2025

Uh oh!

openshift-ci-robot commented Aug 12, 2025

Summary

Changes

Testing

Impact

Uh oh!

bryan-cox commented Aug 13, 2025

Uh oh!

bryan-cox commented Aug 13, 2025

Uh oh!

bryan-cox commented Aug 13, 2025

Uh oh!

Uh oh!

Uh oh!

flavianmissi commented Aug 13, 2025

Uh oh!

openshift-ci bot commented Aug 13, 2025

Uh oh!

xenolinux commented Aug 13, 2025

Uh oh!

sferich888 commented Aug 13, 2025

Uh oh!

bryan-cox commented Aug 13, 2025

Uh oh!

openshift-ci bot commented Aug 13, 2025

Uh oh!

xiuwang commented Aug 14, 2025

Uh oh!

Uh oh!

openshift-ci-robot commented Aug 14, 2025

Summary

Changes

Testing

Impact

Uh oh!

bryan-cox commented Aug 14, 2025

Uh oh!

openshift-ci-robot commented Aug 14, 2025

Uh oh!

openshift-cherrypick-robot commented Aug 14, 2025

Uh oh!

openshift-bot commented Aug 14, 2025

Uh oh!

bryan-cox commented Aug 1, 2025 •

edited

Loading